Another Approach to Comparing Nested and Non-nested Samples

Confronted with the revelation that the 24 possible non-nested samples reuse the same subplots in various fashions, I’m having second thoughts about the methods I proposed to you. It’s not so much that many of the non-nested sample estimates will be correlated with each other (because they share observations), but more that the correlation varies (because the number of observations shared between pairs of samples will differ). In the “d-interval” randomization method I outlined, the null hypothesis of no difference implies that the labeling of the estimates as “nested” or “non-nested” is immaterial and hence the labels can be assigned randomly. But it strikes me now that such a random assignment has the potential of violating the correlation structure of the non-nested samples. To correct for this, the “random” allocation of labels would have to somehow accommodate that structure. This might be possible but it doesn’t strike me as being worth the effort. The bottom line is that I don’t think the labels are truly exchangeable as they’re supposed to be. As a result I’m not sure whether the randomization test I proposed is truly testing the hypothesis of interest. It should be possible to determine whether the correlation structure of the nonnested samples is affecting the estimates that are obtained. For a cursory assessment just plot the 24 estimates obtained from non-nested samples on a number line and see if they cluster in a fashion commensurate with the number of subplots they share in common. A more analytical approach would be to do a Mantel test. Here’s how that test would work. Using the set of 24 non-nested samples from a single plot, obtain 24 estimates of the quantity of interest. Next generate two 24 × 24 distance matrices. The first matrix should contain the absolute value of the difference in the estimates obtained for each pair of samples. The second matrix should reflect the degree of overlap between each pair of samples—larger values indicating less overlap. Since each sample consists of four subplots the overlap distance between two samples can be defined as